Mobile or portable devices often contain a wide feature set to satisfy a variety of use requirements. These devices can include a wide variety of sensors for recording environment data, such as color image camera sensors. Some devices also include depth camera sensors to capture object distance information for a scene. For some applications (for example, augmented reality) depth image information may be used for mapping 3D objects in an environment. The depth camera is typically a separate camera positioned in a different but nearby position than the color camera(s).
Therefore, when a depth camera and color camera are triggered at approximately the same time, the depth camera and camera sensor typically capture images from slightly different viewpoints. The slight differences in viewpoint and other factors such as focal length, resolution, or other differences in camera/sensor properties means that some sort of translation may occur in order to properly apply depth information in a color camera coordinate system. Application of depth information to the color camera coordinate system is desirable because, typically, output of augmented reality applications utilize a color camera image of a scene as the basis for augmentation.
Depth images may be composed of pixels in a 2D coordinate system, where each pixel represents a depth or distance value. The depth information from the depth camera can be mapped to a color camera to assist in augmented reality applications.
However, an augmented reality system typically includes a depth camera physically separate and distinct from the color camera. Therefore, the camera viewpoints for the depth camera and the color camera will not be an identical match. For example, the depth camera may be positioned to the left, right, or some direction relative to the color camera (for example, integrated into a mobile phone or other device).
Because of the parallax effect, the depth map of the depth camera may have a depth value for two points at two different depths where, from the perspective of the color camera, the closer of the two points occludes the further, and hence, the color camera's image captures only the closer of the two points. When attempting to integrate data from the depth map to a color image taken by a color camera, ignoring this above described occlusion can produce errors in output and impact user experience.